Large%2Dscale bot detection for search engines.

PSkip estimating relevance ranking quality from web search clickthrough data.

Identifying %22best bet%22 web search results by mining past user behavior.
In the first part of the paper we consider the problem of dynamically apportioning resources among a set of options in a worst-case on-line framework. The model we study can be interpreted as a broad, abstract extension of the well-studied on-line prediction model to a general decision-theoretic setting. We show that the multiplicative weightupdate rule of Littlestone and Warmuth [20] can be adapted to this model yielding bounds that are slightly weaker in some cases, but applicable to a considerably more general class of learning problems. We show how the resulting learning algorithm can be applied to a variety of problems, including gambling, multiple-outcome prediction, repeated games and prediction of points in R  n  . In the second part of the paper we apply the multiplicative weight-update technique to derive a new boosting algorithm. This boosting algorithm does not require any prior knowledge about the performance of the weak learning algorithm. We also study generalizations of...
Personal name classification in web queries.

Identifying %22best bet%22 web search results by mining past user behavior.
In the first part of the paper we consider the problem of dynamically apportioning resources among a set of options in a worst-case on-line framework. The model we study can be interpreted as a broad, abstract extension of the well-studied on-line prediction model to a general decision-theoretic setting. We show that the multiplicative weightupdate rule of Littlestone and Warmuth [20] can be adapted to this model yielding bounds that are slightly weaker in some cases, but applicable to a considerably more general class of learning problems. We show how the resulting learning algorithm can be applied to a variety of problems, including gambling, multiple-outcome prediction, repeated games and prediction of points in R  n  . In the second part of the paper we apply the multiplicative weight-update technique to derive a new boosting algorithm. This boosting algorithm does not require any prior knowledge about the performance of the weak learning algorithm. We also study generalizations of...
Multistrategy Ensemble Learning Reducing Error by Combining Ensemble Learning Techniques.
Ensemble learning strategies, especially Boosting and Bagging decision trees, have demonstrated impressive capacities to improve the prediction accuracy of base learning algorithms. Further gains have been demonstrated by strategies that combine simple ensemble formation approaches. In this paper, we investigate the hypothesis that the improvement in accuracy of multi-strategy approaches to ensemble learning is due to an increase in the diversity of ensemble members that are formed. In addition, guided by this hypothesis, we develop three new multi-strategy ensemble learning techniques. Experimental results in a wide variety of natural domains suggest that these multi-strategy ensemble learning techniques are, on average, more accurate than their component ensemble learning techniques.
Lessons and Challenges from Mining Retail E%2DCommerce Data.
Abstract. The architecture of Blue Martini Software’s e-commerce suite has supported data collection, transformation, and data mining since its inception. With clickstreams being collected at the application-server layer, high-level events being logged, and data automatically transformed into a data warehouse using meta-data, common problems plaguing data mining using weblogs (e.g. sessionization and conflating multi-sourced data) were obviated, thus allowing us to concentrate on actual data mining goals. We briefly review the architecture and discuss many lessons learned over the last four years and the challenges still facing us. The lessons and challenges are presented across two dimensions: business-level vs. technical, and throughout the data mining lifecycle stages of data collection, data warehouse construction, business intelligence, and deployment. The lessons and challenges are more widely applicable to data mining domains outside retail e-commerce. Keywords: Data mining, data analysis, business intelligence, web analytics, web mining, OLAP,
A Study of AdaBoost with Naive Bayesian Classifiers Weakness and Improvement.

Integrating E%2DCommerce and Data Mining Architecture and Challenges.
We show that the e-commerce domain can provide all the right ingredients for successful data mining. We describe an integrated architecture for supporting this integration. The architecture can dramatically reduce the pre-processing, cleaning, and data understanding effort often documented to take 80% of the time in knowledge discovery projects. We emphasize the  need for data collection at the application server layer (not the web server) in order to support logging of data and metadata that is essential to the discovery process. We describe the data transformation bridges required from the transaction processing  systems and customer event streams (e.g., clickstreams) to the data warehouse. We detail the mining workbench, which needs to provide multiple views of the data through reporting, data  mining algorithms, visualization, and OLAP. We conclude with a set of challenges.
Real world performance of association rule algorithms.
This study compares five well-known association rule algorithms using three real-world datasets and an artificial dataset. The experimental results confirm the performance improvements previously claimed by the authors on the artificial data, but some of these gains do not carry over to the real datasets, indicating overfitting of the algorithms to the IBM artificial dataset. More importantly, we found that the choice of algorithm only matters at support levels that generate more rules than would be useful in practice. For support levels that generate less than 1,000,000 rules, which is much more than humans can handle and is sufficient for prediction purposes where data is loaded into RAM, Apriori finishes processing in less than 10 minutes. On our datasets, we observed super-exponential growth in the number of rules. On one of our datasets, a 0.02% change in the support increased the number of rules from less than a million to over a billion, implying that outside a very narrow range of support values, the choice of algorithm is irrelevant. Categories and Subject Descriptors H.2.8 [Database Management]: Applications -- Data Mining. Keywords Data Mining, Association Rules, Benchmark, Comparisons, Frequent Itemsets, Market Basket Analysis, Affinity Analysis.
Integrating E%2DCommerce and Data Mining Architecture and Challenges
We show that the e-commerce domain can provide all the right ingredients for successful data mining. We describe an integrated architecture for supporting this integration. The architecture can dramatically reduce the pre-processing, cleaning, and data understanding effort often documented to take 80% of the time in knowledge discovery projects. We emphasize the  need for data collection at the application server layer (not the web server) in order to support logging of data and metadata that is essential to the discovery process. We describe the data transformation bridges required from the transaction processing  systems and customer event streams (e.g., clickstreams) to the data warehouse. We detail the mining workbench, which needs to provide multiple views of the data through reporting, data  mining algorithms, visualization, and OLAP. We conclude with a set of challenges.
Constructing X%2Dof%2DN Attributes for Decision Tree Learning.
. While many constructive induction algorithms focus on generating new binary attributes, this paper explores novel methods of constructing nominal and numeric attributes. We propose a new constructive operator, X-of-N. An X-of-N representation is a set containing one or more attribute-value pairs. For a given instance, the value of an X-of-N representation corresponds to the number of its attribute-value pairs that are true of the instance. A single X-of-N representation can directly and simply represent any concept that can be represented by a single conjunctive, a single disjunctive, or a single M-of-N representation commonly used for constructive induction, and the reverse is not true. In this paper, we describe a constructive decision tree learning algorithm, called XofN. When building decision trees, this algorithm creates one X-of-N representation, either as a nominal attribute or as a numeric attribute, at each decision node. The construction of X-of-N representations is carrie...
Lazy Learning of Bayesian Rules.
The naive Bayesian classifier provides a simple and e#ective approach to classifier learning, but its attribute independence assumption is often violated in the real world. A number of approaches have sought to alleviate this problem. A Bayesian tree learning algorithm builds a decision tree, and generates a local naive Bayesian classifier at each leaf. The tests leading to a leaf can alleviate attribute inter-dependencies for the local naive Bayesian classifier. However, Bayesian tree learning still su#ers from the small disjunct problem of tree learning. While inferred Bayesian trees demonstrate low average prediction error rates, there is reason to believe that error rates will be higher for those leaves with few training examples. This paper proposes the application of lazy learning techniques to Bayesian tree induction and presents the resulting lazy Bayesian rule learning algorithm, called Lbr. This algorithm can be justified by a variant of Bayes theorem which supports a weaker conditional attribute independence assumption than is required by naive Bayes. For each test example, it builds a most appropriate rule with a local naive Bayesian classifier as its consequent. It is demonstrated that the computational requirements of Lbr are reasonable in a wide cross-section of natural domains. Experiments with these domains show that, on average, this new algorithm obtains lower error rates significantly more often than the reverse in comparison to a naive Bayesian classifier, C4.5, a Bayesian tree learning algorithm, a constructive Bayesian classifier that eliminates attributes and constructs new attributes using Cartesian products of existing nominal attributes, and a lazy decision tree learning algorithm. It also outperforms, although the result is not statisticall...
KDD%2DCup 2000 Organizers%27 Report Peeling the Onion.
We describe KDD-Cup 2000, the yearly competition in data mining. For the first time the Cup included insight problems in addition to prediction problems, thus posing new challenges in both the knowledge discovery and the evaluation criteria, and highlighting the need to “peel the onion ” and drill deeper into the reasons for the initial patterns found. We chronicle the data generation phase starting from the collection at the site through its conversion to a star schema in a warehouse through data cleansing, data obfuscation for privacy protection, and data aggregation. We describe the information given to the participants, including the questions, site structure, the marketing calendar, and the data schema. Finally, we discuss interesting insights, common mistakes, and lessons learned. Three winners were announced and they describe their own experiences and lessons in the pages following this paper.
Lazy Bayesian Rules A Lazy Semi%2DNaive Bayesian Learning Technique Competitive to Boosting Decision Trees.
Lbr is a lazy semi-naive Bayesian classifier learning technique, designed to alleviate the attribute interdependence problem of naive Bayesian classification. To classify a test example, it creates a conjunctive rule that selects a most appropriate subset of training examples and induces a local naive Bayesian classifier using this subset. Lbr can significantly improve the performance of the naive Bayesian classifier. A bias and variance analysis of Lbr reveals that it significantly reduces the bias of naive Bayesian classification at a cost of a slight increase in variance. It is interesting to compare this lazy technique with boosting and bagging, two well-known state-of-the-art non-lazy learning techniques. Empirical comparison of Lbr with boosting decision trees on discrete valued data shows that Lbr has, on average, significantly lower variance and higher bias. As a result of the interaction of these effects, the average prediction error of Lbr over a range of learning tasks is at...
Stochastic Attribute Selection Committees with Aultiple Boosting Learning More Accurate and More Stable Classifer Committees.

Improving the Performance of Boosting for Naive Bayesian Classification.
. This paper investigates boosting naive Bayesian classification. It first shows that boosting cannot improve the accuracy of the naive Bayesian classifier on average in a set of natural domains. By analyzing the reasons of boosting&#039;s failures, we propose to introduce tree structures into naive Bayesian classification to improve the performance of boosting when working with naive Bayesian classification. The experimental results show that although introducing tree structures into naive Bayesian classification increases the average error of naive Bayesian classification for individual models, boosting naive Bayesian classifiers with tree structures can achieve significantly lower average error than the naive Bayesian classifier, providing a method of successfully applying the boosting technique to naive Bayesian classification. 1 Introduction  For many KDD applications, the prediction (classification) accuracy is the primary concern. Recent studies on the boosting technique have brought...
Classifying Unseen Cases with Many Missing Values.
Handling missing attribute values is an important issue for classifier learning, since missing attribute values in either training data or test (unseen) data affect the prediction accuracy of learned classifiers. In many real KDD applications, attributes with missing values are very common. This paper studies the robustness of four recently developed committee learning techniques, including Boosting, Bagging, Sasc, and SascMB, relative to C4.5 for tolerating missing values in test data. Boosting is found to have a similar level of robustness to C4.5 for tolerating missing values in test data in terms of average error in a representative collection of natural domains under investigation. Bagging performs slightly better than Boosting, while Sasc and SascMB perform better than them in this regard, with  SascMB performing best. Furthermore, we propose a novel voting weight scheme for the committee learning techniques. Although it is very simple, it can improve the robustness of all these ...
An Experimental Evaluation of Integrating Machine Learning with Knowledge Acquisition.
. Machine learning and knowledge acquisition from experts have distinct capabilities that appear to complement one another. We report a study that demonstrates the integration of these approaches can both improve the accuracy of the developed knowledge base and reduce development time. In addition, we found that users expected the expert systems created through the integrated approach to have higher accuracy than those created without machine learning and rated the integrated approach less difficult to use. They also provided favorable evaluations of both the specific integrated software, a system called The Knowledge Factory, and of the general value of machine learning for knowledge acquisition. Keywords: Integrated learning and knowledge acquisition; classification learning; evaluation of knowledge acquisition techniques 1. Introduction  Machine learning  1  and knowledge acquisition from experts provide different and, on the face of it, complementary means of developing knowledge-b...
Stochastic Attribute Selection Committees.
Classifier committee learning methods generate multiple classifiers to form a committee by repeated application of a single base learning algorithm. The committee members vote to decide the final classification. Two such methods, Bagging and Boosting, have shown great success with decision tree learning. They create different classifiers by modifying the distribution of the training set. This paper studies a different approach: Stochastic Attribute Selection Committee learning of decision trees. It generates classifier committees by stochastically modifying the set of attributes but keeping the distribution of the training set unchanged. An empirical evaluation of a variant of this method, namely Sasc, in a representative collection of natural domains shows that the SASC method can significantly reduce the error rate of decision tree learning. On average Sasc is more accurate than Bagging and less accurate than Boosting, although a one-tailed sign-test fails to show that these differences are significant at a level of 0.05. In addition, it is found that, like Bagging, Sasc is more stable than Boosting in terms of less frequently obtaining significantly higher error rates than C4.5 and, when error is raised, producing lower error rate increases. Moreover, like Bagging, Sasc is amenable to parallel and distributed processing while Boosting is not.
Boosting Cost%2DSensitive Trees.
. This paper explores two techniques for boosting cost-sensitive trees. The two techniques differ in whether the misclassification cost information is utilized during training. We demonstrate that each of these techniques is good at different aspects of cost-sensitive classifications. We also show that both techniques provide a means to overcome the weaknesses of their base cost-sensitive tree induction algorithm. 1 Introduction  Cost-sensitive classification deals with situations where different types of incorrect prediction cost differently. The cost of incorrect predictions is more important than the number of incorrect predictions in many real world applications. For example, in medical diagnosis, the cost committed in diagnosing someone as healthy when one has a life-threatening disease is usually considered to be much higher than another type of error---of diagnosing someone as ill when one is in fact healthy. The former type of error is more serious than the latter type of error...
Boosting Trees for Cost%2DSensitive Classifications.
. This paper explores two boosting techniques for cost-sensitive tree classification in the situation where misclassification costs change very often. Ideally, one would like to have only one induction, and use the induced model for different misclassification costs. Thus, it demands robustness of the induced model against cost changes. Combining multiple trees gives robust predictions against this change. We demonstrate that ordinary boosting combined with the minimum expected cost criterion to select the prediction class is a good solution under this situation. We also introduce a variant of the ordinary boosting procedure which utilizes the cost information during training. We show that the proposed technique performs better than the ordinary boosting in terms of misclassification cost. However, this technique requires to induce a set of new trees every time the cost changes. Our empirical investigation also reveals some interesting behavior of boosting decision trees for cost-sensi...
Naive Bayesian Classifier Committees.
. The naive Bayesian classifier provides a very simple yet surprisingly accurate technique for machine learning. Some researchers have examined extensions to the naive Bayesian classifier that seek to further improve the accuracy. For example, a naive Bayesian tree approach generates a decision tree with one naive Bayesian classifier at each leaf. Another example is a constructive Bayesian classifier that eliminates attributes and constructs new attributes using Cartesian products of existing attributes. This paper proposes a simple, but effective approach for the same purpose. It generates a naive Bayesian classifier committee for a given classification task. Each member of the committee is a naive Bayesian classifier based on a subset of all the attributes available for the task. During the classification stage, the committee members vote to predict classes. Experiments across a wide variety of natural domains show that this method significantly increases the prediction accuracy of t...
Scaling Up the Rule Generation of C4.5.
. C4.5 is the most well-known inductive learning algorithm. It can be used to build decision trees as well as production rules. Production rules are a very common formalism for representing and using knowledge in many real-world domains. C4.5 generates production rules from raw trees. It has been shown that the set of production rules is usually both simpler and more accurate than the decision tree from which the ruleset was formed. This research shows that generating production rules from pruned trees usually results in significantly simpler rulesets than generating rules from raw trees. This reduction in complexity is achieved without reducing prediction accuracies. Furthermore, the new approach uses significantly less induction time than the latter. This paper uses experiments in a wide variety of natural domains to illustrate these points. It also shows that the new method scales up better than the old one in terms of ruleset size, the number of rules, and learning time when the tr...
Generating Classifier Commitees by Stochastically Selecting both Attributes and Training Examples.

Constructing conjunctions using systematic search on decision trees.
This paper discusses a dynamic path-based mathod for constructing conjunctions as new attributes for decision tree learning. It searches for conditions (attributevalue pairs) from paths to form new attributes. CAT, a constructive decision tree learning algorithm, which adopts this dynamic path-based method is described. It employs the hypothesis-driven strategy for constructing new attributes, and uses conjunction and negation (implicitly) as its constructive operators. Compared with other hypothesis-driven constructive decision tree learning algorithms such as algorithms of the Fringe family, the new idea of CAT is that it carries out systematic search with pruning over each path of a tree to select conditions for generating a conjunction. Therefore, in CAT, conditions for constructing new attributes are decided dynamically during search. Empirically investigation in a set of artificial and real-world domains shows that CAT can improve the performance of selective decision tree learning in terms of both higher prediction accuracy and lower theory complexity. In addition, it shows some performance advantages over the constructive decision tree learning algorithms that use a fixed path-based method and a fixed rule-based method to construct conjunctions as new attributes. 1 Introduction
Constructing conjunctions using systematic search on decision trees.
This paper discusses a dynamic path-based mathod for constructing conjunctions as new attributes for decision tree learning. It searches for conditions (attributevalue pairs) from paths to form new attributes. CAT, a constructive decision tree learning algorithm, which adopts this dynamic path-based method is described. It employs the hypothesis-driven strategy for constructing new attributes, and uses conjunction and negation (implicitly) as its constructive operators. Compared with other hypothesis-driven constructive decision tree learning algorithms such as algorithms of the Fringe family, the new idea of CAT is that it carries out systematic search with pruning over each path of a tree to select conditions for generating a conjunction. Therefore, in CAT, conditions for constructing new attributes are decided dynamically during search. Empirically investigation in a set of artificial and real-world domains shows that CAT can improve the performance of selective decision tree learning in terms of both higher prediction accuracy and lower theory complexity. In addition, it shows some performance advantages over the constructive decision tree learning algorithms that use a fixed path-based method and a fixed rule-based method to construct conjunctions as new attributes. 1 Introduction


Effects of Different Types of New Attribute on Constructive Induction.
A well-known fundamental limitation of selective induction algorithms is that when tasksupplied attributes are not adequate for, or directly relevant to, describing hypotheses, their performance in terms of prediction accuracy and/or theory complexity is poor. One solution to this problem is constructive induction. It constructs, by using task-supplied attributes, new attributes that are expected to be more appropriate than the task-supplied attributes for describing the target concepts. This thesis focuses on constructive induction with decision trees as the theory description language. It explores: (1) novel approaches to constructing new binary attributes using existing constructive operators, and (2) novel methods of constructing new nominal and new continuous-valued attributes based on a newly proposed constructive operator.  The thesis investigates a fixed rule-based approach to constructing new binary attributes for decision tree learning. It generates conjunctions from producti...
Constructing Nominal X%2Dof%2DN Attributes.
Most constructive induction researchers focus only on new boolean attributes. This paper reports a new constructive induction algorithm, called XofN, that constructs new nominal attributes in the form of X-of-N representations. An X-of-N is a set containing one or more attribute-value pairs. For a given instance, its value corresponds to the number of its attribute-value pairs that are true. The promising preliminary experimental results, on both artificial and real-world domains, show that constructing new nominal attributes in the form of X-of-N representations can significantly improve the performance of selective induction in terms of both higher prediction accuracy and lower theory complexity.  1 Introduction  A well-known elementary limitation of selective induction algorithms is that when task-supplied attributes are not adequate for describing hypotheses, their performance in terms of prediction accuracy and/or theory complexity is poor. To overcome this limitation, constructiv...
